{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 选读：数据并行处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个教程里，我们将学习如何使用数据并行（`DataParallel`）来使用多GPU。\n",
    "\n",
    "PyTorch非常容易的就可以使用GPU，可以用如下方式把一个模型放到GPU上："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda：0\")\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后可以复制所有的张量到GPU上："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mytensor = my_tensor.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`my_tensor.to(device)`返回一个GPU上的`my_tensor`副本，而不是重写`my_tensor`。我们需要把它赋值给一个新的张量并在GPU上使用这个张量。\n",
    "\n",
    "在多GPU上执行前向和反向传播是自然而然的事。然而，PyTorch默认将只是用一个GPU。你可以使用`DataParallel`让模型并行运行来轻易的让你的操作在多个GPU上运行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.DataParallel(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是这篇教程背后的核心，我们接下来将更详细的介绍它。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入和参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "导入PyTorch模块和定义参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "# Parameters and DataLoaders\n",
    "input_size = 5\n",
    "output_size = 2\n",
    "\n",
    "batch_size = 30\n",
    "data_size = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 虚拟数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要制作一个虚拟（随机）数据集，只需实现`__getitem__`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomDataset(Dataset)：\n",
    "\n",
    "    def __init__(self, size, length)：\n",
    "        self.len = length\n",
    "        self.data = torch.randn(length, size)\n",
    "\n",
    "    def __getitem__(self, index)：\n",
    "        return self.data[index]\n",
    "\n",
    "    def __len__(self)：\n",
    "        return self.len\n",
    "\n",
    "rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),\n",
    "                         batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简单模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作为演示，我们的模型只接受一个输入，执行一个线性操作，然后得到结果。然而，你能在任何模型（CNN，RNN，Capsule Net等）上使用`DataParallel`。\n",
    "\n",
    "我们在模型内部放置了一条打印语句来检测输入和输出向量的大小。请注意批等级为0时打印的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module)：\n",
    "    # Our model\n",
    "\n",
    "    def __init__(self, input_size, output_size)：\n",
    "        super(Model, self).__init__()\n",
    "        self.fc = nn.Linear(input_size, output_size)\n",
    "\n",
    "    def forward(self, input)：\n",
    "        output = self.fc(input)\n",
    "        print(\"\\tIn Model： input size\", input.size(),\n",
    "              \"output size\", output.size())\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建一个模型和数据并行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是本教程的核心部分。首先，我们需要创建一个模型实例和检测我们是否有多个GPU。如果我们有多个GPU，我们使用`nn.DataParallel`来包装我们的模型。然后通过`model.to(device)`把模型放到GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(input_size, output_size)\n",
    "if torch.cuda.device_count() > 1：\n",
    "  print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n",
    "  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs\n",
    "  model = nn.DataParallel(model)\n",
    "\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 运行模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以看输入和输出张量的大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data in rand_loader：\n",
    "    input = data.to(device)\n",
    "    output = model(input)\n",
    "    print(\"Outside： input size\", input.size(),\n",
    "          \"output_size\", output.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当我们对30个输入和输出进行批处理时，我们和期望的一样得到30个输入和30个输出，但是若有多个GPU，会得到如下的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2个GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若有2个GPU，将看到"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# on 2 GPUs\n",
    "Let's use 2 GPUs!\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model： input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([5, 5]) output size torch.Size([5, 2])\n",
    "    In Model： input size torch.Size([5, 5]) output size torch.Size([5, 2])\n",
    "Outside： input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3个GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若有3个GPU，将看到："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "Let's use 3 GPUs!\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model： input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside： input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8个GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若有8个GPU，将看到："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "Let's use 8 GPUs!\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside： input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model： input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside： input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataParallel`自动的划分数据，并将作业发送到多个GPU上的多个模型。在每个模型完成作业后，`DataParallel`收集并合并结果返回给你。\n",
    "\n",
    "更多信息，请参考：http://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Spyder)",
   "language": "python3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
